Skip to content

fix: keep combining marks in the default BM25 tokenizer - #13781

Open
Nipuna-Lakruwan wants to merge 2 commits into
deepset-ai:mainfrom
Nipuna-Lakruwan:fix/bm25-combining-marks
Open

Nipuna-Lakruwan wants to merge 2 commits into
deepset-ai:mainfrom
Nipuna-Lakruwan:fix/bm25-combining-marks

Conversation

@Nipuna-Lakruwan

Copy link
Copy Markdown

Related Issues

Proposed Changes:

The default bm25_tokenization_regex of InMemoryDocumentStore is built on \w, which in Python's re does not match combining marks (Unicode categories Mn and Mc). Scripts that write vowels and other signs as marks were therefore cut into fragments: Sinhala "ජනාධිපතිවරයා" became ජන, ධ, පත, වරය, and Tamil, Devanagari, Bengali and vowelled Arabic or Hebrew broke the same way. Those fragments are shared by unrelated words, so BM25 returned documents that don't contain the query word.

This PR:

  • Adds _combining_mark_char_class(), which builds a character class of every combining mark in the Basic Multilingual Plane from unicodedata. The kana voicing marks (U+3099–U+309A) are left out, so CJK keeps its one-token-per-character behaviour from fix: default BM25 tokenizer splits CJK characters for bare-term retrieval #12837.
  • Treats a word as a run of word characters and combining marks.
  • Lets a zero-width joiner or non-joiner (U+200D, U+200C) join two parts of one word. Sinhala uses the joiner in common words such as "ශ්‍රී" (as in Sri Lanka); on its own a joiner is still not a token.

Latin, digits, CJK and punctuation behave exactly as before. For example, the danda (।, ॥) still separates Hindi words.

How did you test it?

  • New unit test test_bm25_tokenization_keeps_combining_marks: Sinhala, Tamil, Bengali, vowelled Arabic, Hindi with dandas, a Sinhala word containing a zero-width joiner, a lone joiner, and decomposed Latin.
  • New retrieval test test_bm25_retrieval_with_combining_mark_script_query: the Tamil query "தமிழ்" returned all three documents before the fix (two of them only shared fragments) and returns only the matching one after it.
  • Both new tests fail on main and pass with this change.
  • test/document_stores/test_in_memory.py and test/components/retrievers: 728 passed, 19 skipped. The existing CJK tests pass unchanged.
  • ruff check and ruff format --check pass.

Notes for the reviewer

  • Building the character class at import takes about 15 ms. I chose to generate it from unicodedata rather than hard-code it, because a hand-written list covering all scripts would be about 190 ranges long. Happy to switch to a hard-coded list, or to a smaller set of scripts, if you prefer.
  • The default regex string is now longer (about 4.8k characters), which shows in to_dict() output when the default is used. If that is a concern, one option is to serialize None for the default and rebuild it on load; I kept this PR minimal and didn't change serialization.
  • Exact-word matching is now consistent with Latin script. For example, Sinhala "පාර්ලිමේන්තුව" no longer partly matches the inflected "පාර්ලිමේන්තුවට", just as "parliament" doesn't match "parliaments". Users who want sub-word matching can still pass their own regex.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@Nipuna-Lakruwan
Nipuna-Lakruwan requested a review from a team as a code owner November 16, 2026 19:04
@Nipuna-Lakruwan
Nipuna-Lakruwan requested review from davidsbatista and removed request for a team November 16, 2026 19:04
@vercel

vercel Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

@Nipuna-Lakruwan is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 16, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@1AbdullahButt 1AbdullahButt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pulled the regex-building code out of the PR head and ran it on its own with the store's lowercase+NFC step. The Sinhala, Tamil, ZWJ, lone-ZWJ and decomposed café cases from the new test all tokenize as expected. Plain Latin/digit text and kana/CJK one-char-per-token are unchanged. I did this on Python 3.10 and 3.12; I didn't run the haystack test suite.

There's one side effect worth deciding on. to_dict() serializes bm25_tokenization_regex as is, so every default InMemoryDocumentStore now writes a pattern of about 4.8 KB into pipeline YAML (4857 chars on 3.12). The pattern also comes from the running interpreter's unicodedata. It differs between 3.10 (Unicode 13.0) and 3.12 (Unicode 15.0), so the same pipeline serializes differently depending on the Python version. A YAML saved on one version also pins that version's mark table. Two options: serialize the default as a sentinel such as None and rebuild it on load, or document this.

One small behaviour change: a combining mark with no base character now becomes a token of its own. "a ́ b" gives ['a', '́', 'b'], while the old regex dropped it. It's probably harmless, but it isn't covered by a test.

Building the class costs about 14 ms at import.

Reviewed with help from Claude Code.

Python's re does not count combining marks as word characters, so the
default bm25_tokenization_regex cut Sinhala, Tamil, Devanagari, Bengali
and vowelled Arabic or Hebrew words into fragments. Treat combining
marks as part of a word, and let a zero-width joiner or non-joiner join
two parts of one word, as in Sinhala conjuncts.
The default tokenization regex is about 4.8 KB and is built from the
running Python's Unicode data, so writing it into to_dict() made pipeline
YAML large and Python-version dependent. Serialize the default as None
and rebuild it on load; passing None also selects the default.

Also require a word to start with a word character, so a combining mark
with no base character is dropped as before instead of becoming a token.
@Nipuna-Lakruwan
Nipuna-Lakruwan force-pushed the fix/bm25-combining-marks branch from 0e967ea to 4fc804e Compare November 17, 2026 21:26
@Nipuna-Lakruwan

Copy link
Copy Markdown
Author

Thanks @1AbdullahButt, this is a really useful review, and checking on both 3.10 and 3.12 caught the version dependence I hadn't thought through. I've pushed 4fc804e, which addresses both points:

Serialization. The default regex is now saved as None and rebuilt on load. to_dict() writes bm25_tokenization_regex: None when the store uses the default, and __init__ treats None as "use the default". Pipeline YAML stays small and identical across Python versions, and the mark table always comes from the Python that loads the pipeline. Custom regexes are still saved as-is. I updated the six tests that expected the full default string in serialized pipelines, and added:

  • test_to_dict_from_dict_round_trip_keeps_default_tokenizer
  • test_init_with_none_regex_uses_default

One thing to note: YAML saved by earlier versions contains the old default regex as a literal string, so those pipelines keep their saved regex until it's removed or set to None. That's the same situation as after the CJK change. I've mentioned it in the release note.

Lone combining marks. A word now has to start with a word character, so "a ́ b" gives ['a', 'b'], as before, and that case is covered in test_bm25_tokenization_keeps_combining_marks.

The branch is rebased on current main. test_in_memory.py, test/components/retrievers and test/tools/test_pipeline_tool.py pass, and the pre-commit hooks pass. Happy to change the approach if maintainers prefer something else.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

InMemoryDocumentStore BM25 tokenizer splits Sinhala, Tamil, Hindi and other words that use combining marks

3 participants